How to: Use EMIT Quality Data¶


There are two quality layers associated with L2A Reflectance Product contained within the L2A Mask file. The mask variable contains 6 binary flag bands (1-5, 8) which should be excluded from analysis, and two data bands (6,7).

The second band_mask variable indicates whether or not any given wavelength of any given pixel is interpolated. Interpolation occurs either due to a focal plane array bad pixel, or from saturation. This data is provided as a packed unsigned integer array with 36 elements.

Note: This How-to guide currently only explains and utilizes the mask data, not the band_mask.

Requirements:

  • A NASA Earthdata Login account is required to download EMIT data
  • Selected the emit_tutorials environment as the kernel for this notebook.
    • For instructions on setting up the environment, follow the instructions in the Prerequisites section of the README.md included in the repository.
  • Downloaded the necessary EMIT files to the ../data/ folder.
    • For download instructions see the Prerequisites section of the README.md included in the repository.

Learning Objectives

  • How to build a mask using the quality flags from an EMIT L2A Mask file
  • How to apply a mask to another EMIT file.

Using EMIT Quality Data¶

Import the required Python libraries.

In [1]:
# Import Packages
import os
import netCDF4 as nc
from osgeo import gdal
import numpy as np
import xarray as xr
import hvplot.xarray
import holoviews as hv
import sys
sys.path.append('../modules/')
import emit_tools

In this guide we will use an EMIT L2A reflectance file as well as an EMIT L2A Mask file. Set the filepaths for each.

In [2]:
fp = '../data/EMIT_L2A_RFL_001_20220903T163129_2224611_012.nc'
fp_mask = '../data/EMIT_L2A_MASK_001_20220903T163129_2224611_012.nc'

The most efficient way to utilize the mask is to apply it before orthorectification because the orthorectified datasets take up more space. To apply a mask using the L2A Mask file, we want to open it, specify which bands to use in construction of a mask, and then apply the mask.

To do this, first take a look at what each band will mask by reading in the sensor_band_parameters group from the mask file as an xarray.dataset then converting to a dataframe.

Note: In the user guide, the bands are indexed as 1-8 not 0-7 as used here.

In [3]:
mask_parameters_ds = xr.open_dataset(fp_mask,engine = 'h5netcdf', group='sensor_band_parameters')
mask_key = mask_parameters_ds['mask_bands'].to_dataframe()
mask_key
Out[3]:
mask_bands
bands
0 Cloud flag
1 Cirrus flag
2 Water flag
3 Spacecraft Flag
4 Dilated Cloud Flag
5 AOD550
6 H2O (g cm-2)
7 Aggregate Flag

The above dataframe shows exactly what each band contained within the file represents/will mask. For this example we will use flags 0,1,3, and 4 to remove any potential clouds and any artefacts caused by the space station. This can be done using the build_mask function from the emit_tools module. This function combines the requested flags into a single mask and returns it as an array.

Select the bands to use.

In [4]:
flags = [0,1,3,4]
flags
Out[4]:
[0, 1, 3, 4]

Now create the mask using the build_mask function.

In [5]:
mask = emit_tools.build_mask(fp_mask,flags)
Flags used: ['Cloud flag' 'Cirrus flag' 'Spacecraft Flag' 'Dilated Cloud Flag']

To see the regions of the unorthocorrected image that will be masked we can plot the mask array.

In [6]:
from matplotlib import pyplot as plt
plt.imshow(mask)
Out[6]:
<matplotlib.image.AxesImage at 0x2eb7e7bdd00>

Now that we have a mask to apply, we can use it as the quality_mask parameter in the emit_xarray function.

Note: If you do not orthocorrect the image, the mask will simply be added as a quality_mask variable.

In [7]:
ds = emit_tools.emit_xarray(fp, ortho=True, quality_mask=mask)
ds
Out[7]:
<xarray.Dataset>
Dimensions:      (latitude: 2009, longitude: 2353, bands: 285)
Coordinates:
  * latitude     (latitude) float64 -39.31 -39.31 -39.31 ... -40.39 -40.4 -40.4
  * longitude    (longitude) float64 -62.51 -62.51 -62.51 ... -61.24 -61.24
    wavelengths  (bands) float32 381.0 388.4 395.8 ... 2.486e+03 2.493e+03
    fwhm         (bands) float32 8.415 8.415 8.415 8.415 ... 8.806 8.807 8.809
    spatial_ref  int32 0
Dimensions without coordinates: bands
Data variables:
    reflectance  (latitude, longitude, bands) float32 nan nan nan ... nan nan
Attributes: (12/38)
    ncei_template_version:             NCEI_NetCDF_Swath_Template_v2.0
    summary:                           The Earth Surface Mineral Dust Source ...
    keywords:                          Imaging Spectroscopy, minerals, EMIT, ...
    Conventions:                       CF-1.63
    sensor:                            EMIT (Earth Surface Mineral Dust Sourc...
    instrument:                        EMIT
    ...                                ...
    southernmost_latitude:             -40.39610428069674
    spatialResolution:                 0.000542232520256367
    spatial_ref:                       GEOGCS["WGS 84",DATUM["WGS_1984",SPHER...
    geotransform:                      [-6.25120945e+01  5.42232520e-04 -0.00...
    day_night_flag:                    Day
    title:                             EMIT L2A Surface Reflectance 60 m V001
xarray.Dataset
    • latitude: 2009
    • longitude: 2353
    • bands: 285
    • latitude
      (latitude)
      float64
      -39.31 -39.31 ... -40.4 -40.4
      long_name :
      Latitude (WGS-84)
      units :
      degrees north
      array([-39.306759, -39.307301, -39.307844, ..., -40.394478, -40.39502 ,
             -40.395562])
    • longitude
      (longitude)
      float64
      -62.51 -62.51 ... -61.24 -61.24
      long_name :
      Longitude (WGS-84)
      units :
      degrees east
      array([-62.512095, -62.511552, -62.51101 , ..., -61.237848, -61.237306,
             -61.236764])
    • wavelengths
      (bands)
      float32
      381.0 388.4 ... 2.486e+03 2.493e+03
      long_name :
      Wavelength Centers
      units :
      nm
      array([ 381.00558,  388.4092 ,  395.81583,  403.2254 ,  410.638  ,
              418.0536 ,  425.47214,  432.8927 ,  440.31726,  447.7428 ,
              455.17035,  462.59888,  470.0304 ,  477.46292,  484.89743,
              492.33292,  499.77142,  507.2099 ,  514.6504 ,  522.0909 ,
              529.5333 ,  536.9768 ,  544.42126,  551.8667 ,  559.3142 ,
              566.7616 ,  574.20905,  581.6585 ,  589.108  ,  596.55835,
              604.0098 ,  611.4622 ,  618.9146 ,  626.36804,  633.8215 ,
              641.2759 ,  648.7303 ,  656.1857 ,  663.6411 ,  671.09753,
              678.5539 ,  686.0103 ,  693.4677 ,  700.9251 ,  708.38354,
              715.84094,  723.2993 ,  730.7587 ,  738.2171 ,  745.6765 ,
              753.1359 ,  760.5963 ,  768.0557 ,  775.5161 ,  782.97754,
              790.4379 ,  797.89935,  805.36176,  812.8232 ,  820.2846 ,
              827.746  ,  835.2074 ,  842.66986,  850.1313 ,  857.5937 ,
              865.0551 ,  872.5176 ,  879.98004,  887.44147,  894.90393,
              902.3664 ,  909.82886,  917.2913 ,  924.7538 ,  932.21625,
              939.6788 ,  947.14026,  954.6027 ,  962.0643 ,  969.5268 ,
              976.9883 ,  984.4498 ,  991.9114 ,  999.37286, 1006.8344 ,
             1014.295  , 1021.7566 , 1029.2172 , 1036.6777 , 1044.1383 ,
             1051.5989 , 1059.0596 , 1066.5201 , 1073.9797 , 1081.4404 ,
             1088.9    , 1096.3597 , 1103.8184 , 1111.2781 , 1118.7368 ,
      ...
             1796.4385 , 1803.8701 , 1811.3008 , 1818.7314 , 1826.1611 ,
             1833.591  , 1841.0206 , 1848.4495 , 1855.8773 , 1863.3052 ,
             1870.733  , 1878.16   , 1885.5869 , 1893.013  , 1900.439  ,
             1907.864  , 1915.2892 , 1922.7133 , 1930.1375 , 1937.5607 ,
             1944.9839 , 1952.4071 , 1959.8295 , 1967.2518 , 1974.6732 ,
             1982.0946 , 1989.515  , 1996.9355 , 2004.355  , 2011.7745 ,
             2019.1931 , 2026.6118 , 2034.0304 , 2041.4471 , 2048.865  ,
             2056.2808 , 2063.6965 , 2071.1123 , 2078.5273 , 2085.9421 ,
             2093.3562 , 2100.769  , 2108.1821 , 2115.5942 , 2123.0063 ,
             2130.4175 , 2137.8289 , 2145.239  , 2152.6482 , 2160.0576 ,
             2167.467  , 2174.8755 , 2182.283  , 2189.6904 , 2197.097  ,
             2204.5034 , 2211.9092 , 2219.3147 , 2226.7195 , 2234.1233 ,
             2241.5269 , 2248.9297 , 2256.3328 , 2263.7346 , 2271.1365 ,
             2278.5376 , 2285.9387 , 2293.3386 , 2300.7378 , 2308.136  ,
             2315.5342 , 2322.9326 , 2330.3298 , 2337.7263 , 2345.1216 ,
             2352.517  , 2359.9126 , 2367.3071 , 2374.7007 , 2382.0935 ,
             2389.486  , 2396.878  , 2404.2695 , 2411.6604 , 2419.0513 ,
             2426.4402 , 2433.8303 , 2441.2183 , 2448.6064 , 2455.9944 ,
             2463.3816 , 2470.7678 , 2478.153  , 2485.5386 , 2492.9238 ],
            dtype=float32)
    • fwhm
      (bands)
      float32
      8.415 8.415 8.415 ... 8.807 8.809
      long_name :
      Full Width at Half Max
      units :
      nm
      array([8.415, 8.415, 8.415, 8.415, 8.417, 8.418, 8.419, 8.421, 8.422,
             8.424, 8.425, 8.426, 8.428, 8.429, 8.431, 8.432, 8.433, 8.435,
             8.436, 8.438, 8.439, 8.44 , 8.442, 8.443, 8.445, 8.446, 8.447,
             8.449, 8.45 , 8.452, 8.453, 8.454, 8.456, 8.457, 8.459, 8.46 ,
             8.461, 8.463, 8.464, 8.466, 8.467, 8.468, 8.47 , 8.471, 8.473,
             8.474, 8.475, 8.477, 8.478, 8.48 , 8.481, 8.482, 8.484, 8.485,
             8.487, 8.488, 8.489, 8.491, 8.492, 8.494, 8.495, 8.496, 8.498,
             8.499, 8.501, 8.502, 8.503, 8.505, 8.506, 8.508, 8.509, 8.51 ,
             8.512, 8.513, 8.515, 8.516, 8.517, 8.519, 8.52 , 8.522, 8.523,
             8.524, 8.526, 8.527, 8.529, 8.53 , 8.531, 8.533, 8.534, 8.536,
             8.537, 8.538, 8.54 , 8.541, 8.543, 8.544, 8.545, 8.547, 8.548,
             8.55 , 8.551, 8.552, 8.554, 8.555, 8.557, 8.558, 8.559, 8.561,
             8.562, 8.564, 8.565, 8.566, 8.568, 8.569, 8.571, 8.572, 8.573,
             8.575, 8.576, 8.578, 8.579, 8.58 , 8.582, 8.583, 8.585, 8.586,
             8.587, 8.589, 8.59 , 8.592, 8.593, 8.594, 8.596, 8.597, 8.599,
             8.6  , 8.601, 8.603, 8.604, 8.606, 8.607, 8.608, 8.61 , 8.611,
             8.613, 8.614, 8.615, 8.617, 8.618, 8.62 , 8.621, 8.622, 8.624,
             8.625, 8.627, 8.628, 8.629, 8.631, 8.632, 8.634, 8.635, 8.636,
             8.638, 8.639, 8.641, 8.642, 8.643, 8.645, 8.646, 8.648, 8.649,
             8.65 , 8.652, 8.653, 8.655, 8.656, 8.657, 8.659, 8.66 , 8.662,
             8.663, 8.664, 8.666, 8.667, 8.669, 8.67 , 8.671, 8.673, 8.674,
             8.676, 8.677, 8.678, 8.68 , 8.681, 8.683, 8.684, 8.685, 8.687,
             8.688, 8.69 , 8.691, 8.692, 8.694, 8.695, 8.697, 8.698, 8.699,
             8.701, 8.702, 8.704, 8.705, 8.706, 8.708, 8.709, 8.711, 8.712,
             8.714, 8.715, 8.716, 8.718, 8.719, 8.721, 8.722, 8.723, 8.725,
             8.726, 8.727, 8.729, 8.73 , 8.732, 8.733, 8.734, 8.736, 8.737,
             8.739, 8.74 , 8.741, 8.743, 8.744, 8.746, 8.747, 8.748, 8.75 ,
             8.751, 8.753, 8.754, 8.755, 8.757, 8.758, 8.76 , 8.761, 8.763,
             8.764, 8.765, 8.767, 8.768, 8.77 , 8.771, 8.772, 8.774, 8.775,
             8.777, 8.778, 8.779, 8.781, 8.782, 8.783, 8.785, 8.786, 8.788,
             8.789, 8.79 , 8.792, 8.793, 8.795, 8.796, 8.797, 8.799, 8.8  ,
             8.802, 8.803, 8.804, 8.806, 8.807, 8.809], dtype=float32)
    • spatial_ref
      ()
      int32
      0
      crs_wkt :
      GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
      semi_major_axis :
      6378137.0
      semi_minor_axis :
      6356752.314245179
      inverse_flattening :
      298.257223563
      reference_ellipsoid_name :
      WGS 84
      longitude_of_prime_meridian :
      0.0
      prime_meridian_name :
      Greenwich
      geographic_crs_name :
      WGS 84
      grid_mapping_name :
      latitude_longitude
      spatial_ref :
      GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
      array(0)
    • reflectance
      (latitude, longitude, bands)
      float32
      nan nan nan nan ... nan nan nan nan
      long_name :
      Surface Reflectance
      units :
      unitless
      array([[[nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              ...,
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan]],
      
             [[nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              ...,
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan]],
      
             [[nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              ...,
      ...
              ...,
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan]],
      
             [[nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              ...,
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan]],
      
             [[nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              ...,
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan],
              [nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)
    • latitude
      PandasIndex
      PandasIndex(Float64Index([  -39.3067591475017, -39.307301380021954, -39.307843612542214,
                     -39.30838584506247,  -39.30892807758273,  -39.30947031010298,
                     -39.31001254262324, -39.310554775143494, -39.311097007663754,
                     -39.31163924018401,
                    ...
                     -40.39068195549418, -40.391224188014434, -40.391766420534694,
                     -40.39230865305495,  -40.39285088557521,  -40.39339311809546,
                     -40.39393535061572,  -40.39447758313597,  -40.39501981565623,
                     -40.39556204817649],
                   dtype='float64', name='latitude', length=2009))
    • longitude
      PandasIndex
      PandasIndex(Float64Index([  -62.5120945327963, -62.511552300276044, -62.511010067755784,
                     -62.51046783523553,  -62.50992560271527,  -62.50938337019502,
                     -62.50884113767476, -62.508298905154504, -62.507756672634244,
                     -62.50721444011399,
                    ...
                     -61.24164373783563,  -61.24110150531537,  -61.24055927279512,
                     -61.24001704027486, -61.239474807754604, -61.238932575234344,
                     -61.23839034271409,  -61.23784811019384,  -61.23730587767358,
                    -61.236763645153324],
                   dtype='float64', name='longitude', length=2353))
  • ncei_template_version :
    NCEI_NetCDF_Swath_Template_v2.0
    summary :
    The Earth Surface Mineral Dust Source Investigation (EMIT) is an Earth Ventures-Instrument (EVI-4) Mission that maps the surface mineralogy of arid dust source regions via imaging spectroscopy in the visible and short-wave infrared (VSWIR). Installed on the International Space Station (ISS), the EMIT instrument is a Dyson imaging spectrometer that uses contiguous spectroscopic measurements from 410 to 2450 nm to resolve absoprtion features of iron oxides, clays, sulfates, carbonates, and other dust-forming minerals. During its one-year mission, EMIT will observe the sunlit Earth's dust source regions that occur within +/-52° latitude and produce maps of the source regions that can be used to improve forecasts of the role of mineral dust in the radiative forcing (warming or cooling) of the atmosphere.\n\nThis file contains L2A estimated surface reflectances and geolocation data. Reflectance estimates are created using an Optimal Estimation technique - see ATBD for details. Reflectance values are reported as fractions (relative to 1).
    keywords :
    Imaging Spectroscopy, minerals, EMIT, dust, radiative forcing
    Conventions :
    CF-1.63
    sensor :
    EMIT (Earth Surface Mineral Dust Source Investigation)
    instrument :
    EMIT
    platform :
    ISS
    processing_version :
    V1.0
    institution :
    NASA Jet Propulsion Laboratory/California Institute of Technology
    license :
    https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/
    naming_authority :
    LPDAAC
    date_created :
    2022-11-14T09:50:54Z
    keywords_vocabulary :
    NASA Global Change Master Directory (GCMD) Science Keywords
    stdname_vocabulary :
    NetCDF Climate and Forecast (CF) Metadata Convention
    creator_name :
    Jet Propulsion Laboratory/California Institute of Technology
    creator_email :
    sarah.r.lundeen@jpl.nasa.gov
    creator_url :
    https://earth.jpl.nasa.gov/emit/
    project :
    Earth Surface Mineral Dust Source Investigation
    project_url :
    https://emit.jpl.nasa.gov/
    publisher_name :
    USGS LPDAAC
    publisher_url :
    https://lpdaac.usgs.gov
    publisher_email :
    lpdaac@usgs.gov
    identifier_product_doi_authority :
    https://doi.org
    flight_line :
    emit20220903t163129_o24611_s000
    time_coverage_start :
    2022-09-03T16:31:29+0000
    time_coverage_end :
    2022-09-03T16:31:41+0000
    software_build_version :
    010603
    product_version :
    01
    history :
    PGE Input files: radiance_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_rdn_b0106_v01.img, pixel_locations_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_loc_b0106_v01.img, observation_parameters_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_obs_b0106_v01.img, surface_model_config=/beegfs/store/emit/ops/repos/emit-sds-l2a/surface/surface_20221020.json
    easternmost_longitude :
    -62.5120945327963
    northernmost_latitude :
    -39.3067591475017
    westernmost_longitude :
    -61.236221412633064
    southernmost_latitude :
    -40.39610428069674
    spatialResolution :
    0.000542232520256367
    spatial_ref :
    GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
    geotransform :
    [-6.25120945e+01 5.42232520e-04 -0.00000000e+00 -3.93067591e+01 -0.00000000e+00 -5.42232520e-04]
    day_night_flag :
    Day
    title :
    EMIT L2A Surface Reflectance 60 m V001

This ds dataset is now orthorectified and the specified mask has been applied. Visualize the output using a plot of a red band (650 nm).

In [8]:
red = np.nanargmin(abs(ds['wavelengths'].data-650)) # Find closes band to 650 - Red
ds.isel(bands=red).hvplot.image(cmap='viridis', aspect = 'equal', frame_width=500, rasterize=True)
Out[8]:

The modified xarray.Dataset can also be saved as a netCDF4 output that can be reopened using the xarray.open_dataset function.

In [ ]:
ds.to_netcdf('../data/example_quality_nc_out.nc')
# Example for Opening 
# ds = xr.open_dataset('../data/example_quality_nc_out.nc')

Contact Info:¶

Email: LPDAAC@usgs.gov
Voice: +1-866-573-3222
Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹
Website: https://lpdaac.usgs.gov/
Date last modified: 01-04-2023

¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I.